Slashing VRAM Usage: Practical Quantization Techniques for Deploying Local LLMs

Running large language models (LLMs) on consumer hardware almost always hits the exact same wall: VRAM limitations. A standard 70-billion-parameter model stored in 16-bit floating-point precision ($\text{FP16}$) requires roughly 70B × 2 bytes = 140 GB of VRAM just to load its weights into memory—completely out of reach for consumer GPUs like an RTX 4090 (24 GB) or even a dual-GPU workstation.

Quantisation is the primary technique that bridges this gap. By reducing the numerical precision of a model's weights and activations, you can shrink VRAM footprints by 50% to 75% with negligible impact on output quality.

Here is a practical, engineering-focused breakdown of how q works, the dominant formats in use, and how to pick the right strategy for local deployment.

1. What Actually Happens During Quantisation?

At its core, quantisation maps high-precision continuous values (like 16-bit floats) to lower-precision discrete integer values (like 8-bit or 4-bit integers).

When a model runs in $\text{FP16}$, every parameter consumes 2 bytes (16 bits) of VRAM. A 7-billion parameter model requires:

Add the Key-Value (KV) cache for long contexts and activation overhead, and you need an 18–24 GB GPU just to run inference comfortably.

Mathematical Mapping

Linear quantization scales floating-point numbers ($x$) into an integer range using a Scale Factor ($S$) and a Zero-Point ($Z$):

To retrieve the approximate original value during matrix multiplication:

Because integer operations ($\text{INT8}/\text{INT4}$) process much faster on modern Tensor Cores and require significantly less memory bandwidth, quantisation reduces both memory consumption and decoding latency.

2. PTQ vs. QAT: Two Distinct Approaches

There are two primary ways to quantise a model:

For consumer deployment, Post-Training Quantisation (PTQ) is the standard choice. You take an existing open-weights model (e.g., Llama, Mistral, Qwen) and convert it directly on local hardware or via community tools without retraining from scratch.

3. The Major Quantisation Formats Compared

Different deployment runtimes rely on different quantisation formats. Choosing the wrong format for your target backend will result in poor GPU utilisation or failed execution.

4. Deep Dive into GGUF: The King of Local Inference

Developed by Georgi Gerganov and the llama. cppcpp community, GGUF (GPT-Generated Unified Format) replaced the legacy GGML format. It stores model architecture metadata, vocabulary, and quantised tensor weights inside a single binary file.

Understanding GGUF K-Quants

GGUF quantisation divides tensor weights into blocks and stores them using lower-precision representations. Different quantisation schemes use different block sizes, scales, and precision strategies to balance model quality against file size and memory usage.

Which GGUF Variant Should You Download?

• Q8_0 (8-bit): Virtually zero perplexity loss compared to FP16. Best used when you have excess VRAM and want maximum accuracy.

• Usually provides a strong quality-to-size balance, with relatively small degradation compared with higher-precision versions on many models.

• Q4_K_M is one of the most widely used choices for local deployment because it offers a strong balance between model size, memory usage, speed, and output quality.

• Q2_K or Q3_K_S (2/3-bit): Aggressive compression. Perplexity degrades noticeably; generally not recommended unless trying to run massive models on severely constrained hardware.

5. VRAM Footprint & Hardware Sizing Matrix

To calculate the VRAM required to load a model in GGUF format, use this working estimation formula:

(Note: The $1.2$ multiplier accounts for PyTorch overhead, CUDA context, and activation memory.)

Realistic Hardware Sizing (7B to 70B Models)

6. Hands-On Execution: Quantising & Serving Locally

Option A: Running Pre-Quantized Models with Ollama

Ollama automates model fetching, CPU/GPU offloading, and GGUF execution under the hood:

# 1. Pull and run a 4-bit quantised Qwen2.5 7B model

ollama run qwen2.5:7b-instruct-q4_K_M

# 2. Check active VRAM usage and model offload status
  
ollama ps

# Expected Terminal Output:

NAME                     ID           SIZE     PROCESSOR    UNTIL
qwen2.5:7b-instruct-q4   a1b2c3d4e5f6  4.7 GB   100% GPU     4 minutes from now
# Check active VRAM usage and model offload status
ollama ps

Option B: Converting HuggingFace Weights to GGUF using llama.cpp

If you fine-tuned a custom model and need to build a GGUF binary manually:

7. Performance & Quality Evaluation: What Do You Lose?

Quantisation does not affect all aspects of a model equally:

1. Perplexity (PPL): Lower bit-widths increase perplexity (a measure of statistical uncertainty). Moving from FP16 to Q5_K_M typically causes an imperceptible rise in PPL ($< 0.05$). Dropping below 3 bits causes PPL to spike sharply.

2. Reasoning & Code Generation: Mathematical tasks, code syntax, and complex multi-step reasoning are the first capabilities to suffer at low bitrates ($\le \text{3-bit}$).

3. Roleplay & General Text Generation: Summarisation, creative writing, and conversational chat remain surprisingly resilient even at 4-bit quantisation.

Summary Strategy

• For Coding & Complex Logic: Use Q5_K_M or Q6_K.

• For General Chat & Retrieval (RAG): Use Q4_K_M.

• For High-Throughput Production: Use AWQ or EXL2 on dedicated Nvidia hardware for maximum generation speed (tokens per second).